Real-time tracking and correlation of microspheres

ABSTRACT

Methods and apparatuses for tracking and correlating particles include an optical detector that captures a first and a second image of the particles. A video detector is used to capture a plurality of video frames of the particles. The video detector captures the video frames of the particles at a rate faster than the rate at which images are captured by the optical detector to track the movement of particles. A first image position of a particle in the first image of the particles is identified, and then the first image position of the particle is correlated to a second image position of the particle in the second image using the plurality of video frames.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/790,327, filed Mar. 15, 2013, the contents of which areincorporated by reference herein.

BACKGROUND OF THE INVENTION

This invention relates to methods and apparatuses for motion trackingand image registration, and more particularly relates to real-timetracking and correlation of particles from images of the particlescaptured at discrete time intervals.

DESCRIPTION OF THE RELATED ART

Given a set of particle images that are captured at discrete timeintervals where the elapsed time between images is too large to resolveindividual random particle movement, identifying the same particle ineach of the images may not be possible. For example, the location of aparticular particle in a first image may, in a second image, be vacantor be occupied by a different particle because within the time betweenwhen the first image was captured and when the second image was capturedthe particles in the image may move. As such, it may be difficult toattribute measurements in two separate images to a common particle.

FIG. 1 is an illustration showing a result of random particle movementbetween two particle images captured at discrete time intervals.Particle image 110 may be an image captured at a time before particleimage 120 is captured. As is evident from the two particle images, 110and 120, the particle 111, in the time between when the first particleimage 110 was captured and when the second particle image 120 wascaptured, has moved from its location on an imaging plane. Without theability to track particle 111 during the time between when the first 110and the second 120 particle images were captured, it may be difficult tocorrelate a measurement corresponding to particle 111 in image 110 to ameasurement corresponding to particle 111 in image 120.

FIG. 2 is an illustration showing changes in particle locations in twoseparate images. Such changes in location may be due to system optics,even if the particles do not move. As with FIG. 1, measurements takenfrom a single particle may come from different areas of a detector.

SUMMARY OF THE INVENTION

Methods and apparatuses for tracking and correlating particles fromimages of the particles captured in discrete time intervals allowsmeasurements to be taken from individual particles among a plurality ofparticles in a plurality of images. In addition to using a first opticaldetector to capture “classification” images (e.g., images directed todetecting optical signatures, such as fluorescence emission) of theparticles, the system may include a second optical detector, hereinreferred to as a video detector, that captures frames at a rate fasterthan the rate at which images are captured by the first opticaldetector. Such a system is able to more closely track the movement ofparticles, and therefore is capable of more accurately identifying thesame particles in a plurality of distinct images taken over a period oftime. As a result, the system may improve the accuracy of data acquiredfrom measurements performed on particles through a plurality of images.

For example, consider a fluorescence optical system for the detection offluorescent particles in a droplet. Due to droplet surfacecharacteristics, the particle movement within the droplet can bevolatile. In order to achieve both a multiplex and high limit ofdetection, multiple “classification” images at different wavelengths mayneed to be captured. When capturing these images in succession with asingle optical detector, several seconds may pass between the time thefirst image and the last image are captured due to moving parts (e.g.,filter wheel, focal position) and/or capture integration time. It can,therefore, become difficult to correlate the positions of the particlesin the first classification image with the positions of the particles ina subsequent classification image (e.g., a second, third, fourth, etc.image) due to the movement of the particles between image captures. Themethods and apparatuses disclosed herein address this problem bytracking and correlating particles in images taken at discrete timeintervals.

A method is disclosed. In one embodiment, the method may includecapturing, using an optical detector, a first and a second image of oneor more particles and capturing, using a video detector, a plurality ofvideo frames of the one or more particles. Furthermore, in someembodiments, the method may include identifying, using the processor, afirst image position of a particle in the first image of the one or moreparticles. In some embodiments, the method may include correlating,using the processor, the first image position of the particle in thefirst image to a second image position of the particle in the secondimage using the plurality of video frames.

In some embodiments, the method may include time stamping each of theimages captured using the optical detector. Furthermore, the method mayinclude time stamping each of the plurality of video frames capturedusing the video detector. The method may include using the time stampsin correlating, using the processor, the first image position of theparticle in the first image to a second image position of the particlein the second image.

In some embodiments, the method may include transforming the identifiedfirst image position in the first image from a position in an imagecoordinate system to a position in a video frame coordinate system toidentify a first video frame position. In addition, the method mayinclude identifying a first video frame of the plurality of video frameswith a time stamp that is closest to the time stamp of the first image.In some embodiments, the method may include identifying a particle witha position in the identified first video frame that is closest to thefirst video frame position. In some embodiments, the method may includetracking the movement of the particle from the first video frame to asecond video frame to locate a second video frame position of theparticle in the second video frame. In addition, the method may includeinverse transforming the second video frame position in the video framecoordinate system to a position in the image coordinate system toidentify a second image position in the second image.

In some embodiments, the method may include capturing, with the videodetector, the plurality of video frames at a frame rate determinedthrough a calibration routine. In some embodiments, the method mayinclude storing the plurality of video frames in a storage device.

In some embodiments, the method may include capturing at least a thirdimage of the one or more particles and correlating the first imageposition of the particle in the first image to a third image position ofthe particle in the third image using the plurality of video frames.

In some embodiments of the method the video detector has a frame ratethat is at least 3 times faster than a frame rate of optical detector.In some embodiments, the video detector has a frame rate that is atleast 3 times faster than a frame rate of optical detector.

In some embodiments, the particles may include two fluorophores, and theoptical detector may be configured to capture the first image and thesecond image at different wavelengths corresponding to the twofluorophores. In addition, in some embodiments, the particles mayinclude three fluorophores, and the optical detector may be configuredto capture the first image, the second image, and the third image atdifferent wavelengths corresponding to the three fluorophores.

An apparatus is also disclosed. In some embodiments, the apparatus mayinclude an imaging region configured to hold one or more particles.Furthermore, the apparatus may include a light radiating deviceconfigured to illuminate the one or more particles in the imagingregion. In some embodiments, the apparatus may include an opticaldetector configured to capture a first and second image of the one ormore particles and a video detector configured to capture a plurality ofvideo frames of the one or more particles. In addition, the apparatusmay include a processor, coupled to the optical detector and the videodetector, and configured to identify a first image position of aparticle in the first image of the one or more particles. The processormay also be configured to correlate the first image position of theparticle in the first image to a second image position of the particlein the second image using the plurality of video frames.

In some embodiments, the processor may be further configured to timestamp each of the images captured with the optical detector and timestamp each of the plurality of video frames captured with the videodetector. In addition, in some embodiments, the processor may be furtherconfigured to transform the identified first image position in the firstimage from a position in an image coordinate system to a position in avideo frame coordinate system to identify a first video frame position.Furthermore, the processor may be configured to identify a first videoframe of the plurality of video frames with a time stamp that is closestto a time stamp of the first image and to identify a particle with aposition in the identified first video frame that is closest to thefirst video frame position. The processor may be further configured totrack the movement of the particle from the first video frame to asecond video frame having a time stamp that is closest to a time stampof the second image to locate a second video frame position of theparticle in the second video frame. In addition, the processor may beconfigured to inverse transform the second video frame position in thesecond video frame from a position in the video frame coordinate systemto a position in the image coordinate system to identify a second imageposition in the second image.

In some embodiments, the apparatus may include a dichroic mirror toseparate light radiating from the light radiating device from lightemitted from the particles in the imaging region. In some embodiments,the apparatus may include an optical component to focus the lightradiated from the particles.

In some embodiments, the apparatus may include a filter positionedbetween the imaging region and the optical detector. The filter may beconfigured to permit light with a first wavelength to pass from theimaging plane, through the filter, and to the optical detector and toreflect or absorb light with a different wavelength than the firstwavelength.

In some embodiments, the video detector may be configured to capturevideo frames at a rate that is at least three times faster than theoptical detector captures images. In some embodiments, the videodetector may be configured to capture video frames at a rate that is atleast ten times faster than the optical detector captures images. Insome embodiments, the processor may be configured to determine, througha calibration routine, the rate at which the video detector capturesvideo frames.

In some embodiments, the processor may be configured to synchronize theoptical detector and the video detector with one another in time usinghardware mechanisms. In some embodiments, the processor may beconfigured to synchronize the optical detector and the video detectorwith one another in time using software mechanisms.

The apparatus of claim 10, where the imaging region is an imaging plane.

The apparatus of claim 10, where the imaging region is athree-dimensional volume.

In certain embodiments, the methods and apparatus disclosed herein useno more than one optical detector and no more than one video detector.

The term “coupled” is defined as connected, although not necessarilydirectly, and not necessarily mechanically.

The terms “a” and “an” are defined as one or more unless this disclosureexplicitly requires otherwise.

The terms “comprise” (and any form of comprise, such as “comprises” and“comprising”), “have” (and any form of have, such as “has” and“having”), “include” (and any form of include, such as “includes” and“including”) and “contain” (and any form of contain, such as “contains”and “containing”) are open-ended linking verbs. As a result, a method ordevice that “comprises,” “has,” “includes” or “contains” one or moresteps or elements possesses those one or more steps or elements, but isnot limited to possessing only those one or more elements. Likewise, astep of a method or an element of a device that “comprises,” “has,”“includes” or “contains” one or more features possesses those one ormore features, but is not limited to possessing only those one or morefeatures. Furthermore, a device or structure that is configured in acertain way is configured in at least that way, but may also beconfigured in ways that are not listed.

The foregoing has outlined rather broadly the features and technicaladvantages of the present disclosure in order that the detaileddescription of the disclosure that follows may be better understood.Additional features and advantages of the disclosure will be describedhereinafter which form the subject of the claims of the disclosure. Itshould be appreciated by those skilled in the art that the conceptionand specific embodiment disclosed may be readily utilized as a basis formodifying or designing other structures for carrying out the samepurposes of the present disclosure. It should also be realized by thoseskilled in the art that such equivalent constructions do not depart fromthe spirit and scope of the disclosure as set forth in the appendedclaims. The novel features which are believed to be characteristic ofthe disclosure, both as to its organization and method of operation,together with further objects and advantages will be better understoodfrom the following description when considered in connection with theaccompanying figures. It is to be expressly understood, however, thateach of the figures is provided for the purpose of illustration anddescription only and is not intended as a definition of the limits ofthe present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The following drawings form part of the present specification and areincluded to further demonstrate certain aspects of the presentinvention. The invention may be better understood by reference to one ormore of these drawings in combination with the detailed description ofspecific embodiments presented herein.

FIG. 1 is an illustration showing a result of random particle movementbetween two particle images captured at discrete time intervals.

FIG. 2 is an illustration showing a result of uniform movement ofparticles between two images captured at discrete time intervals.

FIG. 3 is a schematic block diagram illustrating one embodiment of aparticle imaging apparatus for tracking and correlating particles.

FIG. 4 is a flow chart illustrating one embodiment of a method fortracking and correlating particles.

FIG. 5 is a flow chart illustrating another embodiment of a method fortracking and correlating particles.

FIGS. 6A and 6B are illustrations showing a result of two particleimages of a particle.

FIG. 6C is an illustration showing a result of a plurality of videoframe positions of a particle overlaid on one another.

FIGS. 7A-7B are illustrations showing the result of two particle imagepositions and a plurality of video frame positions of a particleoverlaid on one another.

FIG. 8 is a schematic block diagram illustrating an embodiment of aparticle imaging apparatus for tracking and correlating particles.

FIG. 9 is a flow chart illustrating an embodiment of a method fortracking and correlating particles.

DETAILED DESCRIPTION

Various features and advantageous details are explained more fully withreference to the non-limiting embodiments that are illustrated in theaccompanying drawings and detailed in the following description.Descriptions of well-known starting materials, processing techniques,components, and equipment are omitted so as not to unnecessarily obscurethe invention in detail. It should be understood, however, that thedetailed description and the specific examples, while indicatingembodiments of the invention, are given by way of illustration only, andnot by way of limitation. Various substitutions, modifications,additions, and/or rearrangements within the spirit and/or scope of theunderlying inventive concept will become apparent to those havingordinary skill in the art from this disclosure.

In the following description, numerous specific details are included toprovide a thorough understanding of disclosed embodiments. One ofordinary skill in the art will recognize, however, that embodiments ofthe invention may be practiced without one or more of the specificdetails, or with other methods, components, materials, and so forth. Inother instances, well-known structures, materials, or operations are notshown or described in detail to avoid obscuring aspects of theinvention.

Although certain embodiments are described herein with respect tomicrospheres or beads, it is to be understood that the systems andmethods described herein may also be used with other particles,including microparticles, gold nanoparticles, quantum dots, nanodots,nanoparticles, nanoshells, nanocrystals, droplets, rare-earth metalparticles, magnetic particles, latex particles, cells, micro-organisms,vesicles, organelles, organic matter, non-organic matter, or any otherdiscrete substances known in the art. The particles may be formed from avariety of substances including, latex, polystyrene, agarose, silica,glass, or dextran. In certain embodiments, the particles may be dropletsformed as an emulsion, including for example, a water-in-oil emulsion oran oil-in-water emulsion. In particular embodiments, a droplet volumemay be approximately 1-5 picoliters, 5-10 picoleters; 10-50 picoliters;50-100 picoleters; 100-500 picoleters; 500-1,000 picoleters; 1,000-5,000picoliters; 5,000-10,000 picoleters; 10,000-50,000 picoleters;50,000-100,000 picoliters; 100,000-500,000 picoleters or500,000-1,000,000 picoleters. In particular embodiments, a dropletvolume may be approximately 1-5 femtoliters, 5-10 femtoliters; 10-50femtoliters; 50-100 femtoliters; 100-500 femtoliters; 500-1,000femtoliters; 1,000-5,000 femtoliters; 5,000-10,000 femtoliters;10,000-50,000 femtoliters; 50,000-100,000 femtoliters; 100,000-500,000femtoliters or 500,000-1,000,000 femtoliters. In certain embodiments,the droplet volume may be approximately 1.0-1.5 microliters; 1.5-2.0microliters; 2.0-2.5 microliters; or 2.5-3.0 microliters. The particlesmay have optical properties including color, fluorescence, orchemiluminescence. The particles may serve as vehicles for molecularreactions. Examples of appropriate particles are illustrated anddescribed in U.S. Pat. No. 5,736,330 to Fulton, U.S. Pat. No. 5,981,180to Chandler et al., U.S. Pat. No. 6,057,107 to Fulton, U.S. Pat. No.6,268,222 to Chandler et al., U.S. Pat. No. 6,449,562 to Chandler etal., U.S. Pat. No. 6,514,295 to Chandler et al., U.S. Pat. No. 6,524,793to Chandler et al., and U.S. Pat. No. 6,528,165 to Chandler, which areincorporated by reference as if fully set forth herein. The systems andmethods described herein may be used with any of the particles describedin these patents. In addition, particles for use in method and systemembodiments described herein may be obtained from manufacturers such asLuminex Corporation of Austin, Tex. The terms “beads” and “microspheres”are used interchangeably herein.

In addition, the types of particles that are compatible with the systemsand methods described herein include particles with fluorescentmaterials attached to, or associated with, the surface of the particles.These types of particles, in which fluorescent dyes or fluorescentparticles are incorporated into the particles in order to provide theclassification fluorescence (i.e., fluorescence emission measured andused for determining an identity of a particle or the subset to which aparticle belongs), are illustrated and described in U.S. Pat. No.6,268,222 to Chandler et al. U.S. Pat. No. 6,649,414 to Chandler et al.,and U.S. Pat. No. 7,718,262 to Chandler et al., which are incorporatedby reference as if fully set forth herein. The types of particles thatcan be used in the methods and systems described herein also includeparticles having one or more fluorochromes or fluorescent dyesincorporated into the core of the particles.

Particles that can be used in the methods and systems described hereinfurther include particles that will exhibit one or more fluorescentsignals upon exposure to one or more appropriate light sources.Furthermore, particles may be manufactured such that upon excitation theparticles exhibit multiple fluorescent signals, each of which may beused separately or in combination to identify the particles.

The methods described herein generally include analyzing one or moreimages of particles and processing data measured from the images todetermine one or more characteristics of the particles such as, but notlimited to, numerical values representing the magnitude of fluorescenceemission of the particles at multiple detection wavelengths. Subsequentprocessing of the one or more characteristics of the particles, such asusing one or more of the numerical values to determine a token IDrepresenting the multiplex subset to which the particles belong and/or areporter value representing a presence and/or a quantity of analytebound to the surface of the particles, can be performed according to themethods described in U.S. Pat. No. 5,736,330 to Fulton, U.S. Pat. No.5,981,180 to Chandler et al., U.S. Pat. No. 6,449,562 to Chandler etal., U.S. Pat. No. 6,524,793 to Chandler et al., U.S. Pat. No. 6,592,822to Chandler, U.S. Pat. No. 6,939,720 to Chandler et al., U.S. Pat. No.8,031,918 to Roth, which are incorporated by reference as if fully setforth herein. In one embodiment, the methods described herein can beused in a MagPix® instrument. The MagPix® instrument is a multiplexingplatform with automated image processing software capable of measuringfluorescent intensity of up to 50 optically-distinct populations ofmagnetic beads randomly distributed in an imaging field.

Turning now to the figures, FIG. 3 illustrates one embodiment of aparticle imaging apparatus 300 for tracking and correlating particles.It should be noted that FIG. 3 is not drawn to scale and some elementsof the system are not shown so as to not obscure the system in detail.

According to an embodiment, the particle imaging device 300 may includean imaging plane 302 configured to hold a plurality of particles, and alight radiating device 304 to illuminate the particles on the imagingplane 302. In one embodiment, the light radiating device 304 may be aninfrared illuminator. In some embodiments, the light radiating devicemay be an LED, laser, or multispectral lamp. A dichroic mirror 312 maybe used to separate the light radiating from the light radiating device304 from the light emitted, and thus radiated, from the particles on theimaging plane 302 after being illuminated by the light radiating device304.

The particle imaging device 300 may also include an optical detector 310configured to capture a plurality of images of the particles on theimaging plane 302, and a video detector 306 configured to capture aplurality of video frames of the particles. The light radiating from theparticles on the imaging plane 302 may be focused by an opticalcomponent 316, split by beam splitter 314, and directed to the opticaldetector 310 and the video detector 306. In order to produce morereliable images, for example, the particles may be at least partiallyimmobilized on the imaging plane 302 prior to capturing an image withthe optical detector 310 or a video frame with the video detector 306 ofthe particles.

The particle imaging device 300 may further include a filter 308disposed between the imaging plane 302 and the optical detector 310 tofilter the plurality of images prior to being captured by the opticaldetector 310. In some embodiments, the filter 308 may be configured topermit light with a first wavelength to pass from the imaging plane 302,through the filter 308, and to the optical detector 310, and to reflector absorb light with a different wavelength than the first wavelength.This allows measurements to be performed on particles based on theirfluorescence at a particular wavelength. In one embodiment, each imageof the particles captured by the optical detector 310 may be an image ofthe particles at a particular wavelength. For example, a first image ofthe particles may show only particles that radiate light at a firstwavelength, and a second image of the particles may show only particlesthat radiate light at a second wavelength. In some embodiments, aparticle may radiate light at multiple wavelengths, and therefore mayappear in multiple images of particles filtered at differentwavelengths, such as the first and second images. Because one use of thevideo detector 306 may be to track and resolve the position ofparticles, the video detector may capture video frames of particles atmore than one wavelength. As such, particles may be visible to the videodetector even if they are not visible to the optical detector due tofiltering. Therefore, a particle's position may be tracked using thevideo detector even if the particle does not emit light at a wavelengththat reaches the optical detector.

The particle imaging device 300 may further include a processor 318 thatis coupled to the optical detector 310 and the video detector 306, andthat is configured to synchronize the optical detector 310 with thevideo detector 306. The processor 318 may synchronize, in time, theoptical detector 310 with the video detector 306 by time stamping eachof the plurality of images captured with the optical detector 310, andtime stamping each of the plurality of video frames captured with avideo detector 306. The processor 318 may also be configured tosynchronize the optical detector 310 and the video detector 306 with oneanother in time using hardware, such as an electrical connection thatcarries a synchronizing pulse. In some embodiments, the video detector306 may be the same type of detector as the image detector. However, insome embodiments, the video detector 306 may have less resolution thanthe image detector 310. Furthermore, the video detector 306 may beconfigured to capture video frames at a rate faster than the opticaldetector 310 captures images. By capturing video frames at a rate fasterthan images are captured by the optical detector 310, the video detector306 may allow the processor to more closely track the movement ofparticles between the time instants when images are captured with theoptical detector 310. For example, the video detector 306 may startcapturing video frames when the first image is captured by the opticaldetector 310, and may continue capturing video frames at a faster ratethan the optical detector 310 captures images until the optical detector310 captures its last image of the particles. Because the video frameshave less time between successive frames as compared to images takenwith the optical detector, the movement of particles may be trackedusing the relative locations of the measured particles. The rate atwhich the video frames are captured may be determined through numerousmethods. For example, in one embodiment, the processor 318 may beconfigured to determine the rate at which the video detector 306captures video frames through a calibration routine. Such a calibrationroutine may, for example, acquire video frames at different frame ratesand determine a minimum frame rate that allows for accurate tracking ofparticles given the amount and type of movement of the particles. Theframe rate may then be increased by a predetermined amount to ensurethat particles are accurately tracked. In another embodiment, theprocessor 318 may be configured to determine the rate at which the videodetector 306 captures video frames based on known designcharacteristics, in which case the rate is predetermined to besufficiently fast to accurately capture particle movement. For example,in some embodiments, the video frame rate may be three times the rate atwhich images are captured, which would allow for two video framesbetween two image captures. In some embodiments, the video frame ratemay be ten times as fast as the frame rate of the image detector, whichwould provide nine video frames between two image captures. Larger videoframe rates may allow for better tracking of particles, but may alsorequire more processing and storage resources. The particle imagingdevice 300 may also include a storage device 320 to store the pluralityof video frames captured by the video detector 306 and/or the imagescaptured by the optical detector 310.

The processor 318 may also be configured to identify a first imageposition I_(p1) of a particle X in a first image I₁ captured by theoptical detector 310, and to correlate the first image position I_(p1)of the particle X to a second image position I_(p2) of the particle X ina second image I₂ captured by the optical detector 310 using theplurality of video frames. Locations in an image captured by an opticaldetector 310 may be specified in an image coordinate system andlocations in a video frame captured by a video detector 306 may bespecified in a video frame coordinate system. For example, if the videodetector 306 has less resolution than the optical detector 310,different coordinate systems may be used to describe locations (orparticle positions) in images taken by the two detectors. A particle mayshow up in both images, but may have different coordinates in the twoimages because the two images have different resolution, and thereforedifferent coordinate systems. The processor 318 may be configured totransform the identified first image position I_(p1) in the first imageI₁ from a position in the image coordinate system to a position in thevideo frame coordinate system. By transforming the location coordinatesfrom the image coordinate system to the video frame coordinate system, afirst video frame position F_(p1) may be identified, wherein the firstvideo frame position F_(p1) is the position in a video frame thatcorresponds to the first image position I_(p1) in an image.

In order to correlate particles from separate images captured by theoptical detector 310, the relevant video frames may be analyzed. Forexample, the relevant video frames may be the video frames that arecaptured approximately between the time that the first image I₁ iscaptured by the optical detector 310 and the time that the second imageI₂ is captured. The relationship between the first image I₁ and thesecond image I₂ captured by the optical detector 310 may not be criticalbecause the first and second images, I₁ and I₂ respectively, onlydistinguish that the first image I₁ and the second image I₂ are twoseparate images captured by the optical detector 310. For example, thesecond image I₂ may be the fourth image captured by the optical detector310 after the first image I₁ was captured. As another example, thesecond image I₂ may be captured before the first image I₁ is captured,which may be the case if data is processed after the images are takenand the processor uses an image taken later in time as a starting point.Nevertheless, the most relevant video frame to the first image I₁ may bethe video frame that is captured within the shortest amount of timeeither before or after the first image I₁ was captured. Therefore, theprocessor 318 may be further configured to identify a first video frameF₁, of the plurality of video frames captured by the video detector 306,with a time stamp that is closest to the time stamp of the first imageI₁ captured by the optical detector 310.

In order to track the movement of a particular particle X using thevideo frames, the particle X may be identified in the first video frameF₁. Therefore, the processor 318 may also be configured to identify aparticle Y with a position in the identified first video frame F₁ thatis closest to the first video frame position F_(p1). A particle Y in thefirst video frame F₁ may be the same particle as particle X in the firstimage I₁, therefore particle X and particle Y will hereinafter bereferred to as particle X. To track the movement of particle X from thetime when the first image I₁ was captured to the time when the secondimage I₂ was captured, the processor 318 may be configured to track themovement of particle X from the first video frame F₁ to a second videoframe F₂ to locate a second video frame position F_(p2) of particle X inthe second video frame F₂. In an embodiment, the second video frame F₂may be a video frame with a time stamp that is closest to the time stampof the second image I₂ captured by the optical detector 310, even ifaddition video frames were taken between the first video frame F₁ andthe second video frame F₂. Tracking the movement of particle X may, insome embodiments, include tracking particle X in successive video frameseither in real time or in software post-processing step performed by theprocessor.

With the second video frame position F_(p2) of the particle Xidentified, the processor 318 may be configured to inverse transform thesecond video frame position F_(p2) in the second video frame F₂ from aposition in the video frame coordinate system to a position in the imagecoordinate system to identify a second image position I_(p2) of particleX in the second image I₂. By transforming the location coordinates fromthe video frame coordinate system back to the image coordinate system,the second image position I_(p2) of particle X in the second image I₂may be identified as the position of the particle in the second image I₂that is closest to the second image position I_(p2) for particle X thatwas identified from the inverse transform operation.

FIG. 4 is a flow chart illustrating one embodiment of a method 400 fortracking and correlating particles. A preferred embodiment to implementthe method 400 of FIG. 4 may be the apparatus disclosed in FIG. 3,although one skilled in the art will readily recognize that otherembodiments and apparatuses may also be used to implement the method 400of FIG. 4 without departing from the scope or spirit of this disclosure.At block 402, an optical detector may be synchronized, by a processor,with a video detector. At block 404, the optical detector may be used tocapture a first and a second image of one or more particles. Forexample, FIGS. 6A and 6B are illustrations showing a first image 602 anda second image 604 of a particle 611 captured by an optical detector.The first and second images, 602 and 604 respectively, may be images ofa particle 611 captured by an optical detector at discrete timeintervals. Therefore, 602 may be, for example, the first image of theparticle 611, and 604 may be the second image of the particle 611 takenat a later time than the first image. From FIG. 6A and FIG. 6B it isevident that during the time between when the first image was capturedand when the second image was captured, the particle moved from alocation 611A in the first image 602 to a location 611B in the secondimage 604. As discussed previously, the first image 602 and the secondimage 604 may be images that have been filtered prior to being capturedby the optical detector to show particle 611 at distinct fluorescentwavelengths.

To track the movement of particle 611 as it moves from its location 611Ain the first image 602 to its location 611B in the second image 604, atblock 406, a plurality of video frames of the one or more particles maybe captured with the video detector. For example, FIG. 6C is anillustration showing a plurality of video frame positions 611C-611F ofparticle 611 overlaid on one another to show the movement of particle611 over time. The plurality of video frame positions 611C-611F may showsuccessive video frame positions of particle 611 as captured in distinctvideo frames with the video detector. For example, the video frameposition 611D may be the position of particle 611 in a video framecaptured at a time after the video frame that shows the particle 611 atvideo frame position 611C was captured. Therefore, video frame position611C may be the first video frame position of particle 611 captured withthe video detector, video frame position 611D may be the second videoframe position of particle 611 captured, video frame position 611E maybe the third video frame position of particle 611 captured, and so on.

To identify a particle of interest that may be tracked using a pluralityof video frames, a first image position of a first particle in the firstimage 602 of the one or more particles may, at block 408, be identified.For example, the first particle may be particle 611 from the first image602, and its first image position may be 611A. With the particle ofinterest identified as particle 611, the first image position 611A of afirst particle 611 in the first image 602 may, at block 410, becorrelated to a second image position of the first particle 611 in thesecond image using the plurality of video frames. For example, thesecond image may be the second image 604 of FIG. 6B, the second imageposition may be 611B of FIG. 6B, and the plurality of video frames usedto correlate the first image position 611A from the first image 602 tothe second image position 611B from the second image 604 may be theplurality of video frames that capture particle 611 at the plurality ofvideo frame positions 611C-611F.

FIG. 5 is a flow chart illustrating an embodiment of a method 500 fortracking and correlating particles. More specifically, method 500provides an example of a method that may be used to correlate the firstimage position 611A of a first particle 611 in the first image 602 to asecond image position 611B of the first particle 611 in the second image604 using the plurality of video frames that capture particle 611 at theplurality of video frame positions 611C-611F. In order to ensure thatthe particle being tracked using a plurality of video frames correspondsto particle 611 in the first image 602, the identified first imageposition 611A of a first particle 611 in a first image 602 may, at block502, be transformed from a position in an image coordinate system to aposition in a video frame coordinate system. By doing so, the firstvideo frame position in a video frame that corresponds to the firstimage position 611A of particle 611 in a first image 602 may beidentified.

At block 504, a first video frame with the time stamp that is closest tothe time stamp of the first image 602 may be identified. For example,the video frame that captures particle 611 at position 611C may beidentified as the first video frame with the time stamp that is closestto the time stamp of the first image 602. At block 506, a particle witha video frame position in the identified first video frame that isclosest to the first video frame position may be identified. Forexample, the particle identified at block 506 may be particle 611because the video frame position 611C of particle 611, which wascaptured by the first video frame identified at block 504, is closest tothe first video frame position identified at block 502. Therefore,because the particle identified at block 506 may, in some embodiments,also be the same particle as particle 611, except in the video framecoordinate system as opposed to in the image coordinate system, theparticle identified at block 506 will hereinafter be referred to asparticle 611 as well. This is a desired result because the particle 611being tracked with the plurality of video frames should be the same asthe particle 611 identified in the first image 602 so that theparticle's second image position 611B in a second image 604 may becorrelated with the first image position 611A of the particle 611 in thefirst image 602.

The movement of particle 611 may, at block 508, be tracked from thefirst video frame to a second video frame to locate a second video frameposition of the particle 611 in the second video frame. In someembodiments, particles may be tracked by using absolute locations in asequence of images and assuming that each particle in an image that isclosest to the position of a particle in previous image corresponds tothe same particle. If the sample rate of the video image is fast enough,such assumption should be true. Referring to FIG. 6, the particle 611may be tracked from the video frame that captured particle 611 atposition 611C to the video frame that captured particle 611 at position611F. Therefore, the video frame that captures particle 611 at position611F may be identified as the second video frame and position 611F ofparticle 611 may be the second video frame position. In one embodiment,the second video frame may be a video frame with a time stamp that isclosest to the time stamp of the second image 604 captured by theoptical detector. With the video frame position of particle 611identified within the second video frame, at block 510, the second videoframe position 611F in the second video frame may be inverse transformedfrom a position in the video frame coordinate system to a position inthe image coordinate system to identify a second image position in thesecond image 602. The second image position identified at block 510 as aresult of inverse transforming may or may not be located at the exactsame location as the second image position 611B of particle 611 in thesecond image 604 captured by the optical detector. However, the secondimage position identified at block 510 may be the closest position ofany particle of the second video frame when the positions of theparticles in the second video frame are inverse transformed to obtainimage positions corresponding to the video frame positions of theparticles in the second video frame. Therefore, the second imageposition 611B of particle 611 in the second image 604 may be identifiedusing the plurality of video frames because the second image position611B may be the closest position of any particle in the second image tothe image coordinate system position determined from inversetransforming the second video frame position 611F.

FIG. 7A is an illustration showing two particle image positions and aplurality of video frame positions of a particle overlaid on one anotherto illustrate one embodiment of methods 400 and 500. Particle 711 may beidentified in a first image captured by an optical detector, and thefirst image position of particle 711 may be identified as position 711A.The first image position 711A may be transformed from an imagecoordinate system position to a video frame coordinate system positionto identify a first video frame position. Video frame positions711C-711F may be successively captured with a video detector that may besynchronized with the optical detector. Using the time stamps of theimages and video frames, the video frame that captures a particle atvideo frame position 711C may be identified as the video frame with atime stamp that is closest to the time stamp of the first image thatcaptured particle 711 at first image position 711A. Video frame position711C may be identified as the video frame position closest to thetransform-identified first video frame position, and therefore theparticle located at video frame position 711C may be identified as theparticle to track with the plurality of video frames. This particle isthe same particle as particle 711 from the first image, except that itis in the video frame coordinate system. The movement of the particlemay be tracked successively from the video frames that capture theparticle at video frame positions 711C-711F. The video frames maycontinue to be tracked from the first video frame until a second videoframe, which has a time stamp, among a plurality of video images,closest to a second image captured by the optical detector, isidentified. The tracked particle's second video frame position 711F inthe second video frame may be inverse transformed to identify the secondimage position of particle 711 in the second image captured by theoptical detector. The particle with a position, in the second imagecaptured by the optical detector, closest to the identified second imageposition from the inverse transform operation may be identified as theparticle 711 in the second image, thus correlating the first imageposition 711A of particle 711 in the first image to a second imageposition 711B of the particle in the second image.

FIG. 7B shows an embodiment where, as compared to FIG. 7A and particle711, either the particle 721 moves less between positions 721A and 721Bor where the sample rates of images 721A-B, and video frames 711C-F, arefaster. As seen in FIG. 7B, video frame position 711C is closest toimage position 721A. The particle 721 is then tracked in video framepositions 721C-F. Finally, video frame position 721F is used to identifythe second image position 721B of the particle 721. As such, ameasurement of particle 721 in the first image position 721A and ameasurement of the particle 721 in the second image position 721B can beassociated with the same particle even though particle 721 has movedduring the time between the two images were taken.

FIG. 8 is a schematic block diagram illustrating another embodiment of aparticle imaging apparatus 800 for tracking and correlating microsphereparticles. As with FIG. 3, it should be noted that FIG. 8 is not drawnto scale and some elements of the system are not shown so as to notobscure the system in detail. The imaging plane 802, light radiatingdevice 804, video detector 806, filter 808, optical detector 810, beamsplitter 812, mirror 814, optical component 816, processor 818, andstorage device 820 of particle imaging apparatus 800 function in asimilar manner as the imaging plane 302, light radiating device 304,video detector 306, filter 308, optical detector 310, dichroic mirror312, beam splitter 314, optical component 316, processor 318, andstorage device 320 of particle imaging apparatus 300 in FIG. 3 in thatboth the video detector and image detector can image the same particlessimultaneously. Whereas the particle imaging apparatus 300 of FIG. 3places a dichroic mirror 312 on the side of the optical component 316where the optical detector 310 is located (camera side), the particleimaging apparatus 800 of FIG. 8 utilizes a beam splitter 814 on the sideof the optical component 816 where the light radiating device 804 islocated (object side). Therefore the image detector 810 in FIG. 8 mayhave its own optics separate from the optics of the video detector 806.

FIG. 9 is a flow chart illustrating one embodiment of a method 900 fortracking and correlating microsphere particles. Preferred embodiments toimplement the method 900 of FIG. 9 may be the apparatuses disclosed inFIG. 3 or FIG. 8, although one skilled in the art will readily recognizethat other embodiments and apparatuses may also implement the method 900of FIG. 9 without departing from the scope or spirit of this disclosure.The method 900 may be suitable for scenarios in which two images arecaptured by the optical detector as before, except that the position ofparticles in the second image may not be determinable. For example, if aparticle does not emit fluorescent light at a particular wavelength (oremits an unappreciable amount of light at that wavelength), and thefilter 808 only transmits light having that same wavelength, the imagedetector 810 may not see the particle's fluorescence. The method maybegin at block 902 where a first image and a second image areidentified. The first image and the second image may be images capturedby an optical detector at discrete time intervals. The position ofparticles in the first image may be determinable, but the position ofparticles in the second image may or may not be determinable. At block904, a first image position of a first particle in the first image maybe identified. The first particle for which the first image position wasidentified at block 904 may be a particle for which measurements may besought, and therefore the position of the first particle in multipleimages may be determined in order to collect data from measurementsperformed on multiple images of the first particle. At block 906, thefirst image position may be transformed from a position in an imagecoordinate system to a position in a video coordinate system to identifya first video frame position in a first video frame. The first videoframe may be a video frame captured by a video detector at a timeclosest to the time when the first image was captured by the opticaldetector. As before, a particle may be identified in the first videoframe that has a position in the first video frame that is closest tothe first video frame position identified at block 906 as a result ofthe transform operation. The particle identified in the first videoframe may be the first particle, and the first particle may be trackedin successive video frames that are captured by the video detector. Asdisclosed in connection with FIG. 3 and FIG. 5, the optical detector andthe video detector may be synchronized in time to aid in identifying therelevant video frames to track the movement of the particle. Each of theimages captured by the optical detector and each of the video framescaptured by the video detector may be time stamped to identify thetemporal relationship between images and video frames. The firstparticle may be continuously tracked using video frames that arecontinuously being captured until approximately a time that a secondimage is captured by the optical detector. Although in some embodimentsthe first image and the second image will be taken consecutively, insome embodiments, one or more images may be taken between the first andsecond image.

For the second image captured by the optical detector, two video frameswith two time stamps that are closest to the time stamp of the secondimage may, at block 908, be identified. For each of the two video framesidentified at block 908, the video frame position of a particle that isclosest to the first video frame position in each of the two videoframes may, at block 910, be identified. The video frame position of thetwo particles in each of the two video frames captured at block 908 may,at block 912, be interpolated to determine the first particle's secondvideo frame position in the video frame coordinate system. At block 914,an inverse transform may be applied to the second video frame positionto determine the first particle's image position in the second image.The first particle's image position in the second image may beestablished as the position of a particle in the second image that isclosest to the image position that was a result of the inverse transformoperation at block 914. However, because images taken by the imagedetector may be filtered (only certain wavelengths allowed to reach thedetector) the first particle may not show up in the second image.However, because the location in the second image is known, the factthat the particle does not emit light that is being captured in thesecond image can be known and that information can be attributed to thefirst particle. In some embodiments, if no particle is found within apredetermined radius of an expected location of a particle in an image,it may be determined that that particle does not emit light in thewavelength being detected. By determining the first particle's secondimage position and the first particle's first image position,information about that particle taken in the first and second images maybe correlated (even if the particle does not show up in one of theimages).

The methods and apparatus described herein are generally described asbeing implemented with particles in an imaging plane. However, themethods and apparatus may also be used with particles in other imagingregions. For example, imaging regions may include three dimensionsregions or volumes, such as a droplet or a microwell. In addition,multiple detectors may be used with the methods described herein. Forexample, two or more video detectors placed at different locations maybe used to track the positions of particles moving in three dimensions.That tracking information may then be used to associate measurementstaken by one or more optical detectors, at different times, to aparticular particle.

In some embodiments, tangible computer-readable media, such as CDs,hard-disks, RAM, or Flash memory, for example, may be made, recorded orwritten to with instructions, that when executed by a processor, arecapable of performing the methods described herein.

All of the methods disclosed and claimed herein can be made and executedwithout undue experimentation in light of the present disclosure. Whilethe apparatus and methods of this invention have been described in termsof preferred embodiments, it will be apparent to those of skill in theart that variations may be applied to the methods and in the steps or inthe sequence of steps of the method described herein without departingfrom the concept, spirit and scope of the invention. For example, themethods disclosed herein may be applied in real time as information isbeing gathered, or it can be performed after all measurements have beentaken in a post-processing step. In addition, modifications may be madeto the disclosed apparatus, and components may be eliminated orsubstituted for the components described herein where the same orsimilar results would be achieved. All such similar substitutes andmodifications apparent to those skilled in the art are deemed to bewithin the spirit, scope, and concept of the invention as defined by theappended claims.

Although the present disclosure and its advantages have been describedin detail, it should be understood that various changes, substitutionsand alterations can be made herein without departing from the spirit andscope of the disclosure as defined by the appended claims. Moreover, thescope of the present application is not intended to be limited to theparticular embodiments of the process, machine, manufacture, compositionof matter, means, methods and steps described in the specification. Asone of ordinary skill in the art will readily appreciate from thepresent processes, disclosure, machines, manufacture, compositions ofmatter, means, methods, or steps, presently existing or later to bedeveloped that perform substantially the same function or achievesubstantially the same result as the corresponding embodiments describedherein may be utilized according to the present disclosure. Accordingly,the appended claims are intended to include within their scope suchprocesses, machines, manufacture, compositions of matter, means,methods, or steps.

What is claimed is:
 1. A method, comprising: capturing, using an opticaldetector, a first and a second image of one or more particles;capturing, using a video detector, a plurality of video frames of theone or more particles; identifying, using the processor, a first imageposition of a particle in the first image of the one or more particles;and correlating, using the processor, the first image position of theparticle in the first image to a second image position of the particlein the second image using the plurality of video frames.
 2. The methodof claim 1, further comprising time stamping each of the images capturedusing the optical detector and time stamping each of the plurality ofvideo frames captured using the video detector and using the time stampsin correlating, using the processor, the first image position of theparticle in the first image to a second image position of the particlein the second image.
 3. The method of claim 2, further comprising:transforming the identified first image position in the first image froma position in an image coordinate system to a position in a video framecoordinate system to identify a first video frame position; identifyinga first video frame of the plurality of video frames with a time stampthat is closest to the time stamp of the first image; identifying aparticle with a position in the identified first video frame that isclosest to the first video frame position; tracking the movement of theparticle from the first video frame to a second video frame to locate asecond video frame position of the particle in the second video frame;and inverse transforming the second video frame position in the videoframe coordinate system to a position in the image coordinate system toidentify a second image position in the second image.
 4. The method ofclaim 1, wherein the video detector captures the plurality of videoframes at a frame rate determined through a calibration routine.
 5. Themethod of claim 1, further comprising capturing at least a third imageof the one or more particles, and correlating the first image positionof the particle in the first image to a third image position of theparticle in the third image using the plurality of video frames.
 6. Themethod of claim 1, wherein the video detector has a frame rate that isat least three times faster than a frame rate of optical detector. 7.The method of claim 1, wherein the particles comprise two fluorophores,and the optical detector is configured to capture the first image andthe second image at different wavelengths corresponding to the twofluorophores.
 8. The method of claim 6, wherein the particles comprisethree fluorophores, and the optical detector is configured to capturethe first image, the second image, and the third image at differentwavelengths corresponding to the three fluorophores.
 9. An apparatus,comprising: an imaging region configured to hold one or more particles;a light radiating device configured to illuminate the one or moreparticles in the imaging region; an optical detector configured tocapture a first and second image of the one or more particles; a videodetector configured to capture a plurality of video frames of the one ormore particles; and a processor, coupled to the optical detector and thevideo detector, configured to: identify a first image position of aparticle in the first image of the one or more particles; and correlatethe first image position of the particle in the first image to a secondimage position of the particle in the second image using the pluralityof video frames.
 10. The apparatus of claim 9, wherein the processor isfurther configured to time stamp each of the images captured with theoptical detector and time stamp each of the plurality of video framescaptured with the video detector.
 11. The apparatus of claim 9, whereinthe processor is further configured to: transform the identified firstimage position in the first image from a position in an image coordinatesystem to a position in a video frame coordinate system to identify afirst video frame position; identify a first video frame of theplurality of video frames with a time stamp that is closest to a timestamp of the first image; identify a particle with a position in theidentified first video frame that is closest to the first video frameposition; track the movement of the particle from the first video frameto a second video frame having a time stamp that is closest to a timestamp of the second image to locate a second video frame position of theparticle in the second video frame; and inverse transform the secondvideo frame position in the second video frame from a position in thevideo frame coordinate system to a position in the image coordinatesystem to identify a second image position in the second image.
 12. Theapparatus of claim 9, further comprising: a dichroic mirror to separatelight radiating from the light radiating device from light emitted fromthe particles in the imaging region; and an optical component to focusthe light radiated from the particles.
 13. The apparatus of claim 9,further comprising a filter positioned between the imaging region andthe optical detector, where the filter is configured to: permit lightwith a first wavelength to pass from the imaging plane, through thefilter, and to the optical detector; and reflect or absorb light with adifferent wavelength than the first wavelength.
 14. The apparatus ofclaim 9, wherein the video detector is configured to capture videoframes at a rate that is at least three times faster than the opticaldetector captures images.
 15. The apparatus of claim 9, wherein theprocessor is configured to determine, through a calibration routine, therate at which the video detector captures video frames.
 16. Theapparatus of claim 9, wherein the processor is further configured tosynchronize the optical detector and the video detector with one anotherin time using hardware mechanisms.
 17. The apparatus of claim 9, whereinthe processor is further configured to synchronize the optical detectorand the video detector with one another in time using softwaremechanisms.
 18. The apparatus of claim 9, where the imaging region is animaging plane.
 19. The apparatus of claim 9, where the imaging region isa three-dimensional volume.
 20. A tangible computer-readable mediumcontaining instructions, which when executed by a processor, cause theprocessor to perform the steps comprising: capturing, using an opticaldetector, a first and a second image of one or more particles;capturing, using a video detector, a plurality of video frames of theone or more particles; identifying, using the processor, a first imageposition of a particle in the first image of the one or more particles;and correlating, using the processor, the first image position of theparticle in the first image to a second image position of the particlein the second image using the plurality of video frames.
 21. A tangiblecomputer-readable medium of claim 20 further containing instructions,which when executed by a processor, cause the processor to perform thesteps comprising: transforming the identified first image position inthe first image from a position in an image coordinate system to aposition in a video frame coordinate system to identify a first videoframe position; identifying a first video frame of the plurality ofvideo frames with a time stamp that is closest to the time stamp of thefirst image; identifying a particle with a position in the identifiedfirst video frame that is closest to the first video frame position;tracking the movement of the particle from the first video frame to asecond video frame to locate a second video frame position of theparticle in the second video frame; and inverse transforming the secondvideo frame position in the video frame coordinate system to a positionin the image coordinate system to identify a second image position inthe second image.